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ABSTRACT 


An  important  aspect  of  cost  research  is  the  measurement  of  the  uncertainty  in¬ 
herent  in  the  projection  of  system  cost.  Approaches  to  this  problem  have  in  the  past 
centered  on  intuition  of  the  decision  maker  or  on  sensitivity  analysis.  Only  recently 
have  approaches  utilizing  such  tools  as  s*  'tistical  decision  theory  and  probability  theory 
been  formulated.  The  study  described  here  explores  and  evaluates  three  such  techniques. 
Each  approach  requires: 

(a)  Expression  of  input  estimates  as  probability  distributions  reflecting  un¬ 
certainty. 

(b)  Cost  equations  pertinent  to  the  particular  model. 

Each  approach  generates: 

(a)  Frequency  distributions  for  cost  elements  and  aggregations. 

(b)  Statistical  measures  that  illustrate  the  nature  and  magnitude  of  the  sys¬ 
tem  cost  uncertainty. 

The  approaches  evaluated  are: 

(a)  Derivation  of  Moments  Technique.  This  employs  equations  for  deriving 
the  moments  and  distributions  for  cost  elements  and  aggregations. 

(b)  Monte  Carlo  Technique.  This  is  a  simulation  routine  to  be  used  in  con¬ 
junction  with  RAC’s  Individual  System/OrganizaUon  Cost  model. 

(c)  Symmetric  Approximation.  This  is  a  special  case  of  the  derivation  of 

moments  technique. 

Finally  the  evaluation  aims  at  two  considerations: 

(a)  The  value  of  such  information  on  uncertainty  to  the  Army  decision  maker 

and  analyst. 

(b)  Comparison  of  the  relative  costs  and  advantages  of  each  approach  de¬ 
scribed  above. 
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INTRODUCTION 

The  purpose  of  this  paper  is  to  describe  ongoing  research  for  improving 
and  evaluating  methodologies  for  quantifying  uncertainty  in  cost  analysis.  This 
research  was  conducted  under  the  sponsorship  of  the  Director  of  Cost  Analysis 
of  the  Office  of  the  Comptroller  of  the  Army.  Approaches  to  the  measurement 
of  the  uncertainty  present  in  cost  models  have  in  the  past  centered  at  worst  on 
the  intuition  of  the  decision  maker  and  at  best  on  sensitivity  analysis.  Only 
recently  have  approaches  based  on  statistical-decision  theory  and  probability 
theory  been  attempted.  The  purpose  of  the  study  is  to  explore  and  evaluate 
three  such  approaches.  More  specifically  this  research  attempts  to: 

(a)  Determine  the  value  of  information  on  uncertainty  to  the  Army  de¬ 
cision  maker  and  the  analyst. 

(b)  Compare  relative  costs  and  advantages  of  each  approach  studied. 

(c)  Develop  an  operational  technique  for  use  by  RAC  and  Army  cost 
analysts. 

The  three  approaches  unde,  valuation  are: 

(a)  Derivation  of  Moments  Technique.  This  employs  equations  for  de¬ 
riving  moments  of  cost  elements  and  aggregations  and  distribution  parameters 
of  aggregations. 

(b)  Monte  Carlo  Technique.  This  is  a  simulation  routine  to  be  used  in 
conjunction  with  RAC's  Individual  System/Organization  Cost  (ISOC)  Model.1’* 

(c)  Symmetric  Approximation.  This  is  a  special  case  of  the  derivation 
of  moments  technique. 


COST  UNCERTAINTY  IN  DECISION  MAKING 

The  process  of  decision  making  and  thus  the  fundamental  task  of  the  de¬ 
cision  maker  is  to  choose  among  alternative  ~  urses  of  action.  Often  such 
choice  will  Involve  a  cost-benefit  analysis  of  either  a  formal  or  informal  na¬ 
ture.  With  a  iormai  analysis  as  our  context,  let  us  examine  what  has  been 
given  the  decision  maker  as  costs. 

The  precise  calculation  of  costs  is  not  a  difficult  task.  The  availability 
of  computers  and  cost  models  can  reduce  this  to  routine.  The  calculation 
itself  is  precise  and  can  be  done  rapidly  in  minute  detail. 

The  inputs  to  these  precise  calculations  are,  however,  not  precise;  in 
fact  they  are  often  quite  the  opposite.  The  errors  present  in  each  input  are 
passed  on  to  various  aggregations  until  one  arrives  at  a  total  cost  that  some¬ 
how  involves  each  Individual  error.  Thus,  cost  inputs  are  combined  in  a  com¬ 
puter  model  with  a  multiplicity  of  equations  and  hundreds  of  other  inputs  to 
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form  a  single  estimate  of  total  costs.  This  single  estimate  is  presented  to 
the  decision  maker  with  the  implication  that  although  it  may  not  be  perfect,  it 
is  certainly  the  best  available  estimate,  without  any  statement  as  to  likelihood 
of  occurrence  or  range  of  other  possible  values. 

But  in  the  generation  of  this  aggregate  number,  hundreds  of  imprecise 
numbers  may  have  been  used.  This  fact  is  not  emphasized  to  the  decision 
maker,  nor  does  he  have  a  basis  to  judge  the  precision  of  the  numbers.  Cost 
models  as  currently  conceived  and  used,  then,  uniformly  withhold  from  tne  de¬ 
cision  maker  some  information  that  might  be  vital  to  his  decision.  We  have 
neglected  to  tell  him  all  we  know  about  the  subject  of  the  accuracy  of  our  esti¬ 
mates. 

As  an  example,  let  us  suppose  that  one  of  the  cost  elements  is  a  missile 
airframe.  The  missile  is  not  yet  designed  or  built,  so  no  production-cost  data 
are  available.  A  cost  estimating  relation  (CER)  could  be  used,  for  example, 
to  estimate  that  the  airframe  cost  for  a  new  missile  would  be  about  $32,000. 
The  design  group,  however,  warns  that  some  added  sophistication  might  make 
the  CER  predict  on  the  low  side,  but  that  improvements  in  some  manufacturing 
techniques  promise  lower  costs.  Some  quick  calculations  show  estimates  as 
low  as  $26,000  and  as  high  as  $45,000.  Each  of  these  calculations  is  based  on 
a  set  of  assumptions  concerning  labor  and  overhead  rates,  material  costs  in 
the  future,  and  design  details  not  yet  firm  and  subject  to  some  uncertainty. 
Each  factor  is  considered, and  the  analyst  enters  his  final  estimate  as  $36,000. 

For  each  of  a  multitude  of  other  important  inputs,  similar  decisions  are 
made,  and  a  single  aggregate  cost  is  obtained  as  the  output  of  the  cost  model. 
This  is  somewhat  analogous  to  describing  the  outcome  of  a  dice  throw  by  the 
most  likely  outcome,  7.  It  says  nothing  about  the  almost  as  likely  6’s  and  8’s 
or  of  the  extremes  of  2  and  12  as  shown  by  Fig.  1.  Expanding  this  analogy  to 
eight  dice  each  with  100  sides,  we  begin  to  approach  the  variations  possible 
in  cost  estimates  for  decision  making. 

Obviously  we  are  not  telling  this  hypothetical  decision  maker  all  we  know 
or  all  he  needs  to  know  to  make  a  rational  decision  when  we  report  a  single¬ 
cost  estimate. 

Th°  dangers  of  the  single-cost  estimate  have  been  recognized  for  years, 
and  several  strategies  have  been  developed  for  augmenting  the  analysis  or 
circumventing  the  difficulty.  One  is  isolation  of  the  differences  between  alter¬ 
natives.  Cost  elements  common  to  alternatives  are  estimated  in  a  similar  or 
identical  manner  so  that  only  the  uncertainties  of  the  unique  features  of  alter¬ 
natives  affect  relative  cost.  Another  is  the  use  of  sensitivity  analysis.  Sensi¬ 
tivity  analysis  computes  the  impact  of  errors  in  estimates  and  assumptions, 
identifying  error  sources  important  to  the  choice  of  alternatives.  It  can  pro¬ 
duce  proof  of  insensitivity, or  it  can  provide  evidence  that,  within  a  relevant 
range  of  values,  choice  is  or  is  not  affected  by  estimation  error. 

Returning  to  the  dice  analogy  of  Fig.  1,  this  is  like  saying  7  is  most 
likely,  but  the  number  cculd  go  as  low  as  2  or  as  high  as  12.  If  ihe  possibility 
of  2  or  12  affects  choice,  supplying  that  information  tells  the  decis'on  maker 
something  he  needs  to  make  a  choice.  However,  the  fact  that  2  or  i2  are  the 
least  likely  outcomes  has  nek  been  given.  In  a  similar  manner,  sensitivity 
analysis,  using  judgments  about  the  relevant  range,  produces  an  array  of  num¬ 
bers  that  include  the  analysts’  beliefs  concerning  the  limits  of  the  variables 
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but  excludes  any  knowledge  of  relative  probability.  No  probability  statements 
are  furnished,  and  the  decision  maker  could  be  led  to  believe  that  all  numbers 
in  the  array  are  equally  likely. 

The  choice  of  a  relevant  range  of  values  for  sensitivity  analysis  is  both 
difficult  and  critical  to  its  usefulness.  It  also  reveals  a  dilemma  inherent  in 
the  application  of  sensitivity  analysis.  If  the  analyst  has  evidence  that  the 
value  of  one  of  the  inputs  is  constrained  within  some  upper  and  lower  limits, 
then  this  same  evidence  may  provide  information  on  the  relative  likelihood  of 
particular  values.  If  he  has  only  an  intuitive  belief  about  the  range,  this  too 
may  be  accomplished  by  equally  valid  suppositions  concerning  probabilities 
of  the  values  within  the  range. 

In  the  previous  example  of  the  missile  cost,  if  the  reasoning  that  fixed 
the  relevant  range  of  costs  at  $26,000  to  $45,000  could  yield  information  un 
the  probability  of  occurrence,  then  we  could  supply  the  missing  information 
in  a  form  useful  to  the  decision.  A  possible  statement  might  be  that  there  is 
a  09  percent  probability  that  the  cost  is  less  than  $44,000.  Combined  with  the 
other  elements  of  the  cost  model  with  similar  statements,  we  might  say  that 
although  the  maximum  cost  produced  by  the  sensitivity  analysis  is,  say,  $800 
million,  there  is  a  99  percent  pi^fc^bility  that  the  cost  will  not  exceed  $720 
million.  In  the  sensitivity  analysis,  if  the  $800  million  number  exceeded 
slightly  the  cost  of  another  a’ternative  under  similar  sensitivity  assumptions, 
the  decision  maker  may  have  been  furnished  an  unlikely  cost  set  for  consider¬ 
ation  along  with  all  other  sensitivity  sets,  and  this  set  may  uniquely  favor  a 
different  alternative.  Knowing  just  a  few  simple  probability  statements,  then, 
can  preclude  earnest  consideration  of  cost  estimates  whose  likelihood  is  very 
remote. 

In  short,  although  sensitivity  analysis  is  a  powerful  tool  for  portraying 
the  results  of  estimating  error,  it  leads  to  pondering  highly  unlikely  situations. 
The  same  reasoning  *hat  leads  to  a  determination  of  relevant  range  for  sensi¬ 
tivity  analysis  has  the  potential  of  providing  key  information  in  decision  making. 
The  importance  of  utilizing  this  information  is,  we  believe,  as  great  as  that  of 
providing  cost  analyses  at  all. 


VALUE  OF  UNCERTAINTY  ANALYSIS 

Having  considered  the  shortcomings  of  present  approaches,  let  us  con¬ 
sider  what  benefits  can  be  expected  for  analyst  and  decision  maker  from  use 
of  probability  information. 

Principal  value  for  the  analyst  seems  to  lie  in  the  isolation  of  major 
sources  of  uncertainty  in  the  model.  We  can  construct  for  conceptual  purposes 
the  matrix  shown  in  Fig.  2  illustrating  the  nature  of  inputs. 

Without  probability-distribution  techniques  the  analyst  was  unable  in  the 
past  to  ascertain  into  which  of  the  cells  of  the  matrix  an  ’-out  item  fell.  Now 
applying  probability  techniques  one  can  determine  this  kind  of  information  and 
feed  it  back  to  the  analyst.  Particularly  important  are  the  isolation  of  X ,,-type 
inputs,  t.e.,  those  possessing  highest  uncertainty  and  contribution  to  total  cost. 
Presumably  cost  analysts  would  want  to  work  on  developing  more  accurate 
estimates  of  these  items  than,  say,  Xnl  items  where  total  contribution  is  lowest 
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despite  high  uncertainty.  At  any  rate  this  information  can  tell  the  analyst 
more  clearly  than  ever  before  to  which  inputs  the  total  cost  is  most  sensitive. 


Uncertainty 

Highest - ►  Lowest 

Highest 

*1.  *12  '  '  '  *  In 

Contribution 

*21  *22  ’  *2r> 

to  totol  co»f 

1 

Lowost 

*n,  *„*-■■  *,„ 

Fig.  2 — Inputs  Matrix 

Potential  value  to  the  decision  maker  can  best  be  described  by  some  il¬ 
lustrations.  In  each  illustration  the  frequency  distributions  for  two  alterna¬ 
tives  are  shown.  The  horizontal  axis  in  each  case  represents  the  cost  of  the 
alternative  and  is  increasing  to  the  right.  The  vertical  axis  represents  the 
likelihood  of  occurrence  at  each  cost  level.  Each  of  these  is  a  hypothetical 
case  in  which  equal  effectiveness  or  other  benefits  are  assumed.  The  deci¬ 
sion  maker’s  problem  is  that  of  choosing  the  least-cost  alternative.  If  only 
single-point  cost  estimates  were  provided,  the  decision  maker  would  of  course 
feel  constrained  to  select  the  lower  cost  in  each  case.  However,  Fig.  3  demon¬ 
strates  how  information  provided  by  probability  estimation  could  modify  his 
outlook. 


Fig.  3— -Alternative  Comparison— Case  1 

Most  likely  value  A  >  most  likely  value  & 
variance  A  <  variance  B 

The  peak  of  each  curve  is  at  the  most  likely,  or  modal,  value,  and  that  is, 
in  our  hypothetical  cases,  the  only  cost  total  that  would  be  furnished  a  decision 
maker  in  the  absence  of  uncertainty  analysis.  In  this  example, B  is  expected 
to  be  less  expensive,  but  it  has  a  larger  variance  so  that  higher  costs  are 
more  likely  than  with  A.  Faced  with  this  dilemma,  the  decision  maker  may 
decide  to  avoid  extreme  costs  by  choosing  A  or  to  gamble  on  the  expected 
lower  costs  of  B.  We  cannot  prescribe  for  him,  but  by  providing  the  probabil¬ 
ity  distributions,  we  have  made  known  to  him  a  pitfall  in  selecting  the  “less 
expensive*  alternative. 

Figure  4  illustrates  a  clear-cut  case.  The  largest  possible  cost  of  one 
alternative  is  less  than  the  smallest  possible  cost  of  the  other.  Decision 
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makers  furnished  with  this  information  are  not  likely  to  choose  differently  than 
if  only  point  estimates  were  given.  They  are,  however,  furnished  an  assurance 
that  no  combination  of  likely  errors  of  estimation  would  result  in  a  reversal 
of  relative  costs.  It  should  be  noted  that  this  cannot  be  done  with  sensitivity 
analysis.  There  might  exist  a  combination  of  extreme  values,  each  within  a 
relevant  range,  in  which  the  cost  ranking  is  reversed,  but  sensitivity  analysis 
would  furnish  no  ciue  that  it  is  a  condition  with  negligible  probability  of  occur¬ 
rence. 


Fig.  4 — Altefnotive  Comparison — Cose  2 

Most  likely  value  A  <  most  likely  volue  B 
No  overlap  of  distributions 

In  Fig.  5  a  single-point  estimate  would  furnish  no  basis  for  choice.  With 
mean  values  essentially  the  same  the  decision  maker  must  look  elsewhere  for 
differences.  If  the  probability  distributions  are  furnished,  however,  it  is  ap¬ 
parent  that  the  costs  cannot  be  regarded  as  equal.  If  they  appear  as  shown 
here,  we  presume  that  A  would  be  chosen,  since  it  can  be  better  accommodated 
in  the  budgeting  and  financial  management  system.  The  possibility  of  much 
lower  than  mean  costs  may  have  some  value  too,  and  as  a  result  it  becomes 
impossible  to  speculate  on  choice  outside  specific  cases.  The  important 
thing  is  that  information  helpful  in  the  decision  process  has  been  furnished 
that  would  not  otherwise  be  available. 


The  costs  associated  with  two  alternatives  may  or  may  not  be  signifi¬ 
cantly  different.  Fig.  6  illustrates  two  cost  totals,  each  with  the  same  variance. 
If  variance  is  low,  the  situation  is  as  in  the  second  example;  costs  are 
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different  and  offer  a  basis  for  choice  as  conclusive  as  the  magnitude  of  the 
difference  might  indicate.  Larger  variances  diminish  the  importance  of  the 
most  likely  value  cost  difference.  At  some  combination  of  •closeness”  and  high 
variance  the  cost  difference  may  noi  oe  of  significance  in  the  selection  of  an 
alternative.  In  any  case,  quantification  of  the  probabilities  of  the  differences 
is  a  useful  contribution  to  the  decision  maker's  understanding. 


Fig.  6 — Alternative  Comparison — Case  4 

Most  likely  value  A  <  most  likely  vn|ue  B 
Variance  A  variance  B;  most  likely  value  A  most  likely  value  B 
Variance  A  variance  8,  variance  A  <  variance  A 
Vorianc'-  S  ■  voriance  B 

These  four  examples,  it  should  be  noted,  differ  only  quantitatively.  They 
are  offered  in  these  forms  to  illustrate  the  range  of  possibilities  in  which  use¬ 
ful  information  may  come  from  a  knowledge  of  the  probability  distributions  of 
total  cost. 

DESCRIPTION  OF  TECHNIQUES 

Having  looked  at  shortcomings  of  present  approaches  and  possible  bene¬ 
fits  of  providing  probability  information,  let  us  now  turn  to  a  discussion  of 
three  techniques  for  providing  such  information. 

Each  technique  utilizes  a  different  method  of  producing  the  same  kinds 
of  information.  Each  assumes  that  cost  distributions  are  approximated  by 
beta  distributions.  This  does  not  seem  illogical.  Beta  distributions  are  uni- 
modal,  finite,  and  continuous.  All  of  these  seem  to  be  characteristics  of  most 
cost  items  dealt  with  in  Army  models.  Each  technique  further  assumes  inde¬ 
pendence  of  cost  inputs.  This  could  prove  limiting  in  certain  situations.  A 
number  of  approaches  to  handling  dependency  can  be  applied.  Among  these 
are  incorporation  in  the  model  of  the  functional  relation  between  the  variables, 
statement  of  the  dependent  variable  in  terms  of  auxiliary  variables,  and  use  of 
joint  probability  distributions.  The  problem  of  independence  is  not  felt  to  be 
prohibitive  although  the  systematic  handling  of  input  dependency  is  still  an 
area  for  research.  Finally  the  beta  distribution  can,  as  opposed  to,  for  exam¬ 
ple,  normal  distributions,  describe  various  conditions  of  skewness  and  peak¬ 
edness,  which  is  desirable  in  the  context  of  cost  analysis.  In  each  section 
there  is  a  discussion  of  (a)  the  general  logic  of  each  technique,  (b)  required 
inputs,  and  r")  characteristic  outputs. 
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Finally  for  one  ctf  these  techniques,  the  derivation  of  moments,  an  ex¬ 
tremely  simple  model  is  presented  both  to  illustrate  the  workings  of  this 
model  and  to  give  a  better  understanding  of  what  this  kind  of  technique  can  do. 

Derivation  of  Moments  Technique 

The  principles  underlying  this  technique  are  (a)  a  distribution  may  be 
described  by  its  first  four  statistical  moments,  and  (b)  these  moments  may  be 
combined  according  to  certain  equations  to  give  the  moments  and  the  distribu¬ 
tions  of  various  cost  aggregations. 

Figure  7  is  a  generalized  flow  diagram  of  the  derivation  of  moments  tech 
nique.  We  note  that  this  routine  has  three  major  sections  or  phases.  Phase  I 
accepts  user  input  describing  cost  items  and  converts  this  for  each  cost  item 
to  the  first  four  statistical  moments  of  the  beta  distribution  determined  by  the 
input.  These  sets  of  moments  become  input  to  Phase  II,  along  with  a  descrip¬ 
tion  of  the  relations  between  cost  items  in  a  particular  model.  Phase  II  com¬ 
bines  moments  according  to  these  predetermined  relations.  The  outputs  of 
Phase  n  then  are  the  first  four  moments  of  each  combination  or  aggregation. 
Finally  Phase  IE  computes  the  parameters  and  produces  plots  of  the  beta  dis¬ 
tributions  determined  by  the  aggregate  moments. 


Oar 

*  fc . . ^ 


A 


Fir* I  four 

■msMtt 
of  inprt* 
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Fig.  7 — Derivation  of  Moments  Technique 


This  technique  is  an  adaptation  of  an  approach  first  formulated  by  Sobel 
of  MITRE.3  However,  severe,  changes  that  were  felt  to  be  significant  were 
made  in  the  original  program. 

Perhaps  the  most  important  adaptation  was  the  complete  revision  of 
Phase  I  of  the  program.  The  original  routine  required  a  user  to  input  the  fol¬ 
lowing  four  parameters:  XP,  most  probable  value;  XH,  high  value;  XL,  low 
value;  and  CR,  80  percent  central  range  (defined  to  be  that  range  of  the  vari¬ 
able  which  contains  80  percent  of  the  probability).  The  estimation  of  this  lat¬ 
ter  parameter  was  felt  to  be  extremely  difficult  and  at  best  subject  to  great 
error.  Efforts  toward  modifying  this  requirement  led  to  the  following  new 
approach.  It  is  in  principle  similar  to  that  described  by  Dienemann.*  Instead 
of  allowing  the  analyst  to  specify  any  beta  distribution,  a  fixed  set  of  distribu¬ 
tions,  each  of  which  could  be  described  qualitatively  in  terms  erf  variance  and 
skewness,  was  developed.  Figure  8  illustrates  the  set  of  nine  distributions 
with  which  tests  are  being  run.  In  this  scheme  variances  are  listed  as  high, 
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medium,  and  low,  and  distributions  are  skewed  right,  symmetric,  and  skewed  left. 
Thus  the  user  specifies  the  type  of  distribution  selected  from  the  given  set 
and  any  two  of  the  following  three  parameters:  most  probable  value,  high 
value,  and  low  value.  Since  the  parameters  of  the  distributions  have  been 
fixed,  specification  of  three  of  the  above  parameters  would  overspecify  the 
distribution  and  could  produce  distortions  in  the  representations  of  the  curve. 
Experience  with  analysts  at  RAC  indicates  that  the  analyst  seems  better  able 
to  pick  a  particular  distribution  from  a  fixed  set  than  to  estimate  a  number 
like  the  80  percent  central  range.  In  addition  the  fixed-set  approach  allows 
for  testing  sensitivity  of  results  to  type  of  distribution  ^elected.  Thus  the 
analyst  may  wish  to  see  what  liappens  to  his  cost  figures  if  he  assumes  dis¬ 
tributions  are  skewed  right  as  opposed  to  symmetric.  This  fixed-set  approach, 
then,  is  one  of  the  major  revisions  to  the  original  approach  set  forth  by  Sobel. 


Skewed  left 


Symmetric 


Skewed  right 


Several  additions  have  been  made  to  the  original  formulation.  Previously 
only  the  parameters  of  the  beta  distribution  were  printed  out  by  Phase  in  of 
the  program.  These  consisted  of  the  alpha  and  beta  parameters  along  with  the 
limits  of  a  selected  probability  area  such  as  upper  and  lower  10  percent  tails. 
However,  a  routine  has  been  added  to  plot  the  frequency  distribution  of  differ¬ 
ent  cost  aggregations.  This  provides  a  graphic  display  of  the  possible  cost 
spread  along  with  the  likelihood  of  different  cost  levels.  This  type  of  display 
is  particularly  effective  in  the  comparison  of  alternatives.  Finally,  several 
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Personnel  x  pay  =  PI 
F/W  x  F/W  Main  =  P2 
PI  +  P2  =  Total 

Fig.  9— Format  for  Derivation  of  Moments  Technique  for  Tv.'o-Equation  Model 

additions  of  an  operational  nature,  such  as  the  automatic  linking  of  phases  of 
the  program,  have  been  made. 

It  might  be  helpful  as  a  means  of  further  illustrating  this  technique  to 
trace  its  operation  in  the  context  of  an  extremely  simple  example.  Figure  9 
illustrates  the  flow  diagram  for  a  cost  model  consisting  of  the  following  two 
equations: 

(a)  PERSONNEL  (expressed  in  number  of  men)  x  PAY  (expressed  in 
dollars  per  man) 

(b)  FIXED-WING  AIRCRAFT  (expressed  in  number  of  planes)  x  FIXED- 
WING  MAINTENANCE  (expressed  in  dollars  per  plane) 

As  inputs  to  Phase  I  the  user  specifies  the  most  probable,  high,  and  low  values 
in  addition  to  type  of  distribution.  Phase  I  then  outputs  the  first  four  moments 
of  the  beta  distribution  determined  by  each  input  set.  Thus  this  model  will  out¬ 
put  four  sets  of  moments. 

These  four  sets  of  moments  now  become  input  to  Phase  II.  Along  with 
these  go  the  following  three  equations  that  make  up  our  model: 

Pi  ■  Personnel  x  pay 
P2  *  F/W  x  K/W  maintenance 
Total  coat  •  Pi  ♦  P2 

Phase  II  then  outputs  three  sets  of  four  moments.  The  first  set  represents 
the  product  of  personnel  and  pay;  the  second,  the  product  of  fixed-wing  craft 
and  fixed-wing  maintenance;  and  the  third,  the  sum  of  these  two  products. 
These  three  sets  now  enter  Phase  in.  The  results  of  this  phase  will  be  the 
distributions  determined  }y  the  three  sets.  They  illustrate  the  cost  spread 
implied  in  this  model  along  with  the  relative  frequency  of  each  cost  level.  In 
addition,  various  parameters  such  as  mean, variance,  limits  of  probability 
areas,  etc.,  can  be  readily  obtained. 

To  further  clarify  operations  of  this  technique,  Fig.  10  illustrates  the 
flow  for  the  same  two-equation  model  with  sample  Phase  I  input  and  Phase 
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III  output.  Thus  for  personnel,  XP  =  10,000  men,  XH  -=  13,000  men, 

XL  =  8,000  men,  and  type  of  distribution  is  8.  The  selection  of  type  8  distri¬ 
bution  from  our  list  of  nine  implies  a  distribution  of  low  variance  with  sym¬ 
metry.  Apparently  the  analyst  here  feels  that  a  level  of  10,000  men  is  fairly 
certain  (indicated  by  picking  low  variance)  and  whatever  deviation  did  occur 
would  be  likely  to  occur  in  either  direction  (indicated  by  pickirg  symmetric 
distribution). 

Outputs  for  Phases  I  and  II  are  not  shown  since  their  addition  would  add 
no  real  understanding.  Also  only  the  frequency  curves  were  shown  as  Phase 
HI  output  because  of  space  limitations.  This  simple  example  then  illustrates 
the  operation  of  the  derivation  of  moments  technique. 


Monte  Carlo  Technique 


Figure  11  is  a  general  flow  diagram  of  the  Monte  Carlo  technique.  This 
technique  is  quite  similar  to  one  formulated  by  Dienemann  of  the  RAND  Cor¬ 
poration.4  The  principle  underlying  this  technique  is  that  if  many  iterations 
of  the  same  cost  model  can  be  computed,  while  changing  the  value  of  the  mod¬ 
el’s  inputs  in  accordance  with  the  given  distributions  on  each  iteration,  the 
relative  frequency  of  various  cost  levels  can  be  computed,  plotted,  and  statis¬ 
tically  analyzed.  The  speed  of  the  computer  enables  us  to  do  this.  Thus  in 
this  technique  we  construct,  compute,  and  analyze  a  large  number  of  iterations 
of  the  same  cc^t  model.  These  three  functions  provide  a  useful  framework  for 
examining  the  flow  chart  of  Fig.  11. 

The  Monte  Carlo  subroutine  functions  as  the  constructor  of  the  inputs  to 
the  iterations.  It  does  this  by  selecting  sets  of  values  for  the  cost  inputs.  The 
subroutine  accepts  from  the  user  the  most  probable,  high,  and  low  values  along 
with  a  type  of  distribution  for  each  cost  input.  These  are  the  same  parameters 
required  by  the  derivation  of  moments  technique.  Based  on  the  type  of  distri¬ 
bution  and  using  a  random  number  generator,  the  subroutine  selects  a  value 
for  each  cost  input.  This  set  of  values  will  be  used  for  this  particular  itera¬ 
tion  only.  These  values  are  next  combined  in  the  ISOC  subroutine. 

The  ISOC  subroutine  is  an  adaptation  of  the  ISOC  computerized  cost  model 
formulated  by  RAC.153  It  wiil  accept  (a)  a  different  set  of  values  selected  oy  the 
Monte  Carlo  subroutine  for  each  iteration  and  (b)  the  relations  between  the 
cost  inputs.  Since  these  relations  hold  for  each  iteration,  they  need  only  be 
input  during  the  first  Iteration,  With  these  inputs  and  relations  the  ISOC  model 
will  compute  cost  aggregations  including  total  cost  and  store  the  results  of  the 
iterations  for  selected  aggregations.  Having  stored  this  information  the  model 
will  ask  whether  a  sufficient  number  erf  Iterations  have  been  run.  A  number  of 
methods  for  choosing  this  can  be  applied.  If  more  iterations  are  required, 
control  is  sent  back  to  the  Monte  Carlo  subroutine  for  the  generation  of  another 
set  of  values.  If  the  required  iterations  have  been  executed,  control  passes  to 
the  final  phase  of  the  technique,  the  analysis  subroutine. 

The  analysis  subroutine  (a)  prepares  a  frequency  distribution  for  each 
selected  aggregation  and  (b)  comfwtes  such  statistical  measures  as  mean  and 
variance  for  the  aggregations.  The  output  of  this  subroutine  thus  closely  re¬ 
sembles  that  of  P!ase  in  of  the  derivation  of  moments  technique.  In  this  ap¬ 
proach,  however,  the  distributions  are  not  plotted  from  the  parameters  erf  a 
beta  distribution  but  from  the  frequency  erf  occurrence  of  various  cost  levels. 
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Symmetric -Approximation  Technique 

Figure  12  illustrates  the  symmetric-approximation  technique.  This 
method  differs  from  the  derivation  of  moments  technique  only  in  the  calcula¬ 
tion  of  cost-input  moments.  Since  the  remaining  two  phases  are  identical,  we 
need  only  focus  attention  on  Phase  I  of  this  technique.  The  heart  of  the  tech¬ 
nique  lies  in  the  four  equations  shown  in  Fig.  12.  These  are  equations  for 
approximating  the  moments  of  beta  distributions.  Note  that  the  third  and  fourth 
moments  are  zero.  This  implies  (a)  the  distribution  is  symmetric,  and  (b,  the 
distribution  has  the  same  flatness  as  a  normal  curve.  These  equations  were 
first  formulated  in  the  development  of  program  evaluation  and  review  tech¬ 
niques  (PERT)  when  tlm  time  needed  to  complete  a  project  was  assumed  to  be 
beta-distributed.  These  equations  do  not  require  the  user  to  input  the  type  of 
distribution  he  feels  characterizes  a  cost  input.  Thus  the  user  inputs  most 
probable,  high,  and  low  values  only,  and  Phase  I  computes  the  first  and  second 
moments.  These  are  then  input  along  with  cost  input  relations  to  Phase  II— 
identical  to  Phase  II  of  the  derivation  of  moments  technique.  Once  again  mo¬ 
ments  of  aggregations  are  outputs  of  Phase  II  and  feed  into  a  Phase  III  identi¬ 
cal  to  that  of  the  derivation  of  moments  technique. 

Several  reasons  Prompted  selection  of  the  technique  for  evaluation.  Such 
equations  have  been  used  to  develop  cost  ranges  for  cost-effectiveness  studies. 
The  author  also  wished  to  find  whether  results  were  significantly  affected  by 
attempts  to  measure  skewness  and  peakedness.  Finally  the  author  was  inter¬ 
ested  in  measuring  at  least  qualitatively  the  validity  or  accuracy  of  these  ap¬ 
proximating  equations. 


FUTURE  STUDY 

As  was  mentioned  at  the  start,  this  paper  describes  an  ongoing  study  at 
RAC.  This  means  that  what  has  been  presented  here  is  not  a  final  product. 
Rather,  it  represents  a  progress  report  designed  to  develop  interest  in  these 
kinds  of  technt-  »s  and  possibly  to  provide  a  basis  for  dialogue  with  others 
working  in  this  ueld  of  uncertainty  quantification.  Also  it  extended  the  oppor¬ 
tunity  for  the  author  to  puli  together  the  pieces  of  the  study,  to  evaluate  efforts 
to  date,  and  to  give  direction  to  future  research.  The  following  are  several 
areas  that  will  be  investigated  in  the  future. 

Comparison  of  Techniques 

Consistent  with  objectives,  first  effort  will  go  to  a  comparison  of  relative 
costs  and  advantages  of  each  technique.  Thus  far  comparison  has  been  limited 
since  one  of  the  models  is  not  yet  completely  operational.  However,  some  con¬ 
clusions  that  seem  significant  have  been  reached; 

1.  The  Monte  Carlo  approach  seems  more  flexible  since  one  could  allow 
distributions  other  than  beta  as  inputs  with  minimal  programming  clauses. 

2.  Monte  Carlo  also  allows  use  of  existing  cost  models  in  its  total  scheme 
whereas  the  other  two  currently  accept  the  format  of  the  derivation  of  moments 
technique  only. 


3.  Time  required  per  iteration  in  the  Monte  Carlo  technique  could  prove 
to  be  prohibitive  for  large-scale  models.  The  other  two  seem  quite  reasonable 
in  the  consumption  of  computer  time. 

4.  The  symmetric  approximation  technique  seems  inadequate  as  it  takes 
no  account  of  any  skewness.  Analysts  generally  believe  cost  distributions  to 
be  skewed  right.  If  this  proves  widespread,  the  error  induced  by  this  approxi¬ 
mation  will  certainly  prove  more  important  than  the  simplification  offered. 

These  statements  are  inconclusive  and  provide  only  the  beginnings  of  a 
basis  for  evaluation.  Once  all  techniques  are  operational,  the  evaluation  of 
their  relative  merits  and  costs  is  the  item  of  first  priority  for  the  future. 

Sample  Analysis  for  Large-Scale  Cost  Model 

The  approach  to  date  has  been  operations  rather  than  applications  oriented. 

Thus  the  models  developed  were  test  cases  containing  considerably  fewer 
equations  than  a  typical  Army  application.  Plans  now  are  to  apply  the  uncer¬ 
tainty  techniques  to  a  division  cost  model  recently  developed  by  the  Cost  Re¬ 
search  Division  of  the  Office  of  the  Comptroller  of  the  Army.  In  order  to  do 
this  for  the  derivation  of  moments  technique,  plans  are  being  formulated  for  a 
-outine  that  will  automatically  link  the  ISOC  format  to  that  of  the  derivation  of 
moments.  This  will  give  the  derivation  of  moments  technique  the  capability 
of  processing  existing  cost  models.  This  could  be  a  determining  factor  in  the 
L  ,1  selection  of  techniques.  At  any  rate  a  large-scale  model  consisting  of 
100  to  200  cost  equations  is  the  next  target  for  application  of  these  uncertainty 
analysis  techniques. 

Development  of  ‘Best1*  Set(s)  of  Distributions 

The  third  area  of  study  is  a  determination  of  the  number  and  types  of  in¬ 
put  distributions  that  would  comprise  a  “best”  set  for  use  by  analysts.  It  is, 
however,  not  clear  that  a  single  set  could  be  chosen  to  handle  all  models  in  all 
cases.  Perhaps  instead  a  different  set  of  distributions  would  be  appropriate 
for  different  kinds  of  cost  models.  A  paper5  now  in  process  of  publication 
introduces  the  concept  of  “families”  of  Army  cost  models.  This  paper  pro¬ 
poses  thai  the  cost  structure  of  each  family  such  as  aircraft  systems  models 
has  certain  standard  characteristics  even  though  individual  models  may  vary 
in  some  details.  The  usefulness  of  such  a  framework  for  the  definition  of  best 
sets  will  be  studied.  A  final  point  pertaining  to  this  selection  is  that  criteria 
for  choice  do  not  seem  obvious  at  all.  At  present  we  conceive  of  a  method  of 
testing  these  families  of  models  with  various  distribution  types,  discussing  > 

results  of  these  tests  with  other  analysts,  and  perhaps  coming  to  agreement 
on  a  set  for  different  models.  Thus  at  this  point  in  time  the  area  oi  definition 
of  best  sets  of  distributions  seems  an  important  though  extremely  difficult  task. 

CONCLUSION 

Although  much  research  remains  to  be  done  before  these  techniques  can 
be  properly  evaluated  and  implemented,  there  is  cause  for  optimism  about  their 
potential  usefulness  to  both  decision  maker  and  analyst.  Techniques  such  as 
ese  will  not  eliminate  uncertainty  from  Input  data  but  can  provide  a  powerful 
tool  for  decision  making  by  measuring  uncertainty  in  a  way  never  before  possible. 


18 


REFERENCES 

1.  John  J.  Surroeier,  ‘An  Individual  System/Organlzation  Cost  Model,*  RAC-TP-183, 
Vol  I,  ‘Concept  and  Application,*  Research  Analysis  Corporation,  31  Aug  65. 

2.  Jodie  T.  Allen,  ‘An  Individual  System/Organization  Cost  Model,*  RAC-TP-183, 

Vol  n,  ‘Computer  Program  Design  and  Operation,*  Research  Analysis  Corporation, 
31  Aug  65. 

3.  S.  A.  Sobel,  ‘A  Computerized  Technique  To  Express  Uncertainty  in  Advanced  Sys¬ 
tem  Cost  Estimates,*  The  MITRE  Corporation,  Sep  63. 

4.  Paul  F.  Dlcnemann,  ‘Estimating  Cost  Uncertainty  Using  Monte  Carlo  Techniques,* 
The  RAND  Corporation,  Jan  66. 

5.  A.  D.  Stament,  ‘An  Individual  System/Organlzation  Cost  Model,*  RAC-TP-183, 

Vol  IV,  ‘Review  of  Selected  Applications, *  Research  Analysis  Corporation,  in 
preparation. 


